System and method for providing preferential i/o treatment to devices that host a critical virtual machine

ABSTRACT

Quality of service is provided to prioritized VMs and applications, based on the varied quality of different shared computing resources. Each VM or application has an associated priority. A quality rating is dynamically assigned to each shared computing resource. Requests for shared computing resources made by specific VMs or applications are received. For each specific received request, the current priority of the requesting VM or application is identified. In response to each received request, a specific shared computing resource is assigned to the specific requesting VM or application. This assignment is made based on the current priority of the requesting VM or application and the current quality rating of the shared computing resource, thereby providing quality of service to the requesting VM or application corresponding to its current priority.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of U.S. patentapplication Ser. No. 15/364,667, filed on Nov. 30, 2016, entitled“System and Method for Providing Preferential I/O Treatment to Devicesthat Host a Critical Virtual Machine,” which is set to issue on Jul. 2,2019 as U.S. Pat. No. 10,338,950, and which is a continuation of U.S.patent application Ser. No. 13/720,566, filed on Dec. 19, 2012, entitled“Providing Optimized Quality of Service to Prioritized Virtual Machinesand Applications Based on Quality of Shared Resources,” which issued onDec. 6, 2016 as U.S. Pat. No. 9,515,899, all of which are incorporatedby reference herein in their entirety and for all purposes.

TECHNICAL FIELD

This disclosure pertains generally to computing device virtualization,and more specifically to providing optimized quality of service toprioritized virtual machines and applications based on the varyingquality of shared computing resources.

BACKGROUND

Clusters are groups of computers that use groups of redundant computingresources in order to provide continued service when individual systemcomponents fail. More specifically, clusters eliminate single points offailure by providing multiple servers, multiple network connections,redundant data storage, etc. Clustering systems are often combined withstorage management products that provide additional useful features,such as journaling file systems, logical volume management, multipathinput/output (I/O) functionality, etc. For example, some storagemanagement products such as Veritas Volume Manager and DynamicMultipathing support multipathed storage devices, in which a virtualdisk device is made available to initiators of I/O, wherein multiplephysical paths exist between the virtual disk and the underlyingphysical storage.

In a high-availability clustering system, the failure of a server (or ofa specific computing resource used thereby such as a network adapter,storage device, etc.) is detected, and the application that was beingrun on the failed server is automatically restarted on another computingsystem. This process is called “failover.” The high availabilityclustering system can also detect the failure of the application itself,and failover the application to another node. In effect, the highavailability clustering system monitors applications, the servers theapplications run on, and the resources used by the applications, toensure that the applications remain highly available. Clusters can beused to provide applications to customers according to service levelagreements guaranteeing varying levels of availability.

Virtualization of computing devices can be employed in high availabilityclustering and in other contexts. One or more virtual machines (VMs orguests) can be instantiated at a software level on physical computers(host computers or hosts), such that each VM runs its own operatingsystem instance. Just as software applications, including serverapplications such as databases, enterprise management solutions ande-commerce websites, can be run on physical computers, so too can theseapplications be run on virtual machines. A high availability cluster ofVMs can be built, in which the applications being monitored by the highavailability clustering system run on and are failed over between VMs,as opposed to physical servers.

In some virtualization scenarios, a software component often called ahypervisor can act as an interface between the guests and the hostoperating system for some or all of the functions of the guests. Inother virtualization implementations, there is no underlying hostoperating system running on the physical, host computer. In thosesituations, the hypervisor acts as an interface between the guests andthe hardware of the host computer, in effect functioning as the hostoperating system, on top of which the guests run. Even where a hostoperating system is present, the hypervisor sometimes interfacesdirectly with the hardware for certain services. In some virtualizationscenarios, the host itself is in the form of a guest (i.e., a virtualhost) running on another host.

A hypervisor receives requests for resources from VMs, and allocatesshared resources such as CPU, memory, I/O bandwidth, I/O channels,storage, performance boosting cache, replication links, etc. In astorage management environment, multipathed storage can also be sharedbetween VMs or hosts. Although conventional hypervisors can allocatedifferent shares of the resources to different VMs, conventionalhypervisors treat all available resources of a given type (e.g., CPU,memory and I/O channels) as being similar and operating in essentiallythe same way. This limits the extent to which varying quality of servicecan be provided to different VMs and applications based on theirpriority or the underlying service level agreements with customers.

It would be desirable to address this issue.

SUMMARY

Quality of service is provided to prioritized VMs or other applicationson a computer, based on the varied quality of different shared computingresources. Each VM or application has a priority, which can indicate thequality of service it is to be provided with relative to other VMs orapplications. Shared computing resources are accessible by multiple VMsor applications. Shared computing resources can be shared among multipleVMs or applications running on a single computer, for example tofacilitate virtualization. A quality rating is assigned to each sharedcomputing resource. In some embodiments, a quality rating comprises asingle quantification of the overall quality of a specific sharedcomputing resource. Assigned quality ratings can also quantify aplurality of qualitative factors concerning specific types of sharedcomputing resources, or specific instances of shared computingresources. Shared computing resources can be periodically evaluated inorder to determine current quality ratings based on their currentstatus. The current quality ratings are then assigned to the sharedcomputing resources.

Requests for shared computing resources made by specific VMs orapplications are received. For example, the received requests can be inthe form of requests for shared computing resources made by specific VMsto a hypervisor, for example to access a virtual disk. For each specificreceived request, the priority of the requesting application isidentified. Identifying the priority of a requesting application canfurther comprise identifying the specific application that made therequest for shared computing resources, for example from a tag in therequest itself. Where the received request is in the form of an IOoperation, the shared computing resource can be identified by thetargeted LUN. In response to each received request, a specific sharedcomputing resource is assigned to the specific requesting application.This assignment is made based on the priority of the requestingapplication and the quality rating of the shared computing resource,thereby providing quality of service to the requesting applicationcorresponding to its priority. In some embodiments, informationdocumenting usage of shared computing resources by applications overtime is logged for future reference.

In one embodiment, received requests for shared computing resourcescomprise requests made by specific applications to initiate IOoperations targeting a specific storage device. In this case, the sharedcomputing resources are in the form of a plurality of queues foraccessing the specific storage device. Each queue is configured toaccess the specific storage device with a different level of priority.In this embodiment, requesting applications are assigned a specific oneof the queues for processing the IO operation.

In another embodiment, received requests for shared computing resourcesare in the form of requests made by specific applications to amultipathing component, in order to access a multipathed storage device.In this embodiment, the shared computing resources can comprise aplurality of queues for accessing a specific one of multiples paths tophysical storage, wherein assigning quality ratings further comprisesassigning a specific level of priority to each queue. The sharedcomputing resources can instead further comprise the plurality of pathsto physical storage, in which case a quality rating is assigned to aspecific path as a quantification of its quality.

The features and advantages described in this summary and in thefollowing detailed description are not all-inclusive, and particularly,many additional features and advantages will be apparent to one ofordinary skill in the relevant art in view of the drawings,specification, and claims hereof. Moreover, it should be noted that thelanguage used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter, resort to theclaims being necessary to determine such inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary network architecture in whicha quality of service manager can be implemented, according to someembodiments.

FIG. 2 is a block diagram of a computer system suitable for implementinga quality of service manager, according to some embodiments.

FIG. 3 is a block diagram of the operation of a quality of servicemanager, according to some embodiments.

FIG. 4 is a block diagram of a shared storage architecture utilized by aquality of service manager, according to some embodiments.

FIG. 5 is a block diagram of the modules of a quality of servicemanager, according to some embodiments.

FIG. 6 is a block diagram illustrating a specific use case of theoperation of a quality of service manager, according to someembodiments.

The Figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an exemplary network architecture100 in which a quality of service manager 101 can be implemented. In theillustrated network architecture 100, client systems 103A, 103B and103N, as well as servers 105A and 105N, are communicatively coupled to anetwork 107. A quality of service manager 101 is illustrated as residingon servers 105A and 105N, but in other embodiments the quality ofservice manager 101 can reside on more, fewer or different computers 210as desired. In FIG. 1, server 105A is further depicted as having storagedevices 160A(1)-(N) directly attached, and server 105N is depicted withstorage devices 160B(1)-(N) directly attached. Servers 105A and 105N arealso connected to a SAN fabric 170 which supports access to storagedevices 180(1)-(N) by servers 105A and 105N, and so by client systems103A-N via network 107. Intelligent storage array 190 is also shown asan example of a specific storage device accessible via SAN fabric 170.In other embodiments, shared storage is implemented using FC and iSCSI(not illustrated) instead of (or in combination with) a SAN fabric 170.

Many different networking technologies can be used to provideconnectivity from each of client computer systems 103A-N to network 107.Some examples include: LAN, WAN and various wireless technologies.Client systems 103A-N are able to access applications and/or data onserver 105A or 105N using, for example, a web browser or other clientsoftware (not shown). This enables client systems 103A-N to runapplications from an application server 105 and/or to access data hostedby a storage server 105 or one of storage devices 160A(1)-(N),160B(1)-(N), 180(1)-(N) or intelligent storage array 190.

Although FIG. 1 illustrates three clients 103A-N and two servers 105A-Nas an example, in practice many more (or fewer) computers can bedeployed. In one embodiment, the network 107 is in the form of theInternet. Other networks 107 or network-based environments can be usedin other embodiments.

FIG. 2 is a block diagram of a computer system 210 suitable forimplementing a quality of service manager 101. The clients 103 andservers 105 illustrated in FIG. 1 can be in the form of computers 210such as the one illustrated in FIG. 2. As illustrated, one component ofthe computer system 210 is a bus 212. The bus 212 communicativelycouples other components of the computer system 210, such as at leastone processor 214, system memory 217 (e.g., random access memory (RAM),read-only memory (ROM), flash memory), an input/output (I/O) controller218, an audio output interface 222 communicatively coupled to anexternal audio device such as a speaker system 220, a display adapter226 communicatively coupled to an external video output device such as adisplay screen 224, one or more interfaces such as serial ports 230,Universal Serial Bus (USB) receptacles 230, parallel ports (notillustrated), etc., a keyboard controller 233 communicatively coupled toa keyboard 232, a storage interface 234 communicatively coupled to atleast one hard disk 244 (or other form(s) of magnetic media), a host busadapter (HBA) interface card 235A configured to connect with a FibreChannel (FC) network 290, an HBA interface card 235B configured toconnect to a SCSI bus 239, an optical disk drive 240 configured toreceive an optical disk 242, a mouse 246 (or other pointing device)coupled to the bus 212 e.g., via a USB receptacle 228, a modem 247coupled to bus 212, e.g., via a serial port 230, and a network interface248 coupled, e.g., directly to bus 212.

Other components (not illustrated) may be connected in a similar manner(e.g., document scanners, digital cameras, printers, etc.). Conversely,all of the components illustrated in FIG. 2 need not be present. Thecomponents can be interconnected in different ways from that shown inFIG. 2.

The bus 212 allows data communication between the processor 214 andsystem memory 217, which, as noted above may include ROM and/or flashmemory as well as RAM. The RAM is typically the main memory into whichthe operating system and application programs are loaded. The ROM and/orflash memory can contain, among other code, the Basic Input-Outputsystem (BIOS) which controls certain basic hardware operations.Application programs can be stored on a local computer readable medium(e.g., hard disk 244, optical disk 242) and loaded into system memory217 and executed by the processor 214. Application programs can also beloaded into system memory 217 from a remote location (i.e., a remotelylocated computer system 210), for example via the network interface 248or modem 247. In FIG. 2, the quality of service manager 101 isillustrated as residing in system memory 217. The workings of thequality of service manager 101 are explained in greater detail below inconjunction with FIG. 3.

The storage interface 234 is coupled to one or more hard disks 244(and/or other standard storage media). The hard disk(s) 244 may be apart of computer system 210, or may be physically separate and accessedthrough other interface systems.

The network interface 248 and or modem 247 can be directly or indirectlycommunicatively coupled to a network 107 such as the Internet. Suchcoupling can be wired or wireless.

FIG. 3 illustrates the operation of a quality of service manager 101,according to some embodiments. FIG. 3 illustrates an instance of aquality of service manager 101 running at a hypervisor 307 level on eachone of multiple physical computers 210 of a cluster 300, wherein one ormore VMs 305 run on each physical computer 210, as discussed in greaterdetail below. It is to be understood that in different embodiments, thefunctionalities of the quality of service manager 101 can reside on aserver 105, client 103, or be distributed between multiple computersystems 210, including within a cloud-based computing environment inwhich the functionality of the quality of service manager 101 isprovided as a service over a network 107. It is to be understood thatalthough a quality of service manager 101 is illustrated in FIG. 3 as asingle entity, the illustrated quality of service manager 101 representsa collection of functionalities, which can be instantiated as a singleor multiple modules as desired. It is to be understood that the modulesof the quality of service manager 101 can be instantiated (for exampleas object code or executable images) within the system memory 217 (e.g.,RAM, ROM, flash memory) of any computer system 210, such that when atleast one processor 214 of the computer system 210 processes a module,the computer system 210 executes the associated functionality. As usedherein, the terms “computer system,” “computer,” “client,” “clientcomputer,” “server,” “server computer” and “computing device” mean oneor more computers configured and/or programmed to execute the describedfunctionality. Additionally, program code to implement thefunctionalities of the quality of service manager 101 can be stored oncomputer-readable storage media, such that when the program code isloaded into computer memory 217 and executed by at least one processor214 of the computer system 210, the computer system 210 executes theassociated functionality. Any form of non-transitory computer readablemedium can be used in this context, such as magnetic or optical storagemedia. As used herein, the term “computer readable medium” does not meanan electrical signal separate from an underlying physical medium.

FIG. 3 illustrates a cluster 300 instantiated in the context of aclustering and storage system 301 in conjunction with a virtualizationenvironment 311. For efficiency of illustration and explanation, boththe clustering and storage system 301 and the virtualization environment311 are illustrated as centralized components. It is to be understoodthat, in practice, both the clustering and storage system 301 and thevirtualization environment 311 contain components that are distributedthroughout the cluster 300. FIG. 3 illustrates a cluster 300 comprisingtwo host computers 210, each running two VMs 305 (guests). The VMs 305are provided by the virtualization environment 311. Although FIG. 3depicts only two physical computers 210, it is to be understood that indifferent embodiments clusters 300 can contain more (or fewer) computersystems 210 as desired. Each VM 305 runs on a physical host computer210, for example of the type illustrated in FIG. 2. One or more VMs 305can run on each host 210 (two VMs 305 per host 210 are illustrated inFIG. 3, although one VM 305 or more than two VMs 305 can run on a singlehost 210). In turn, one or more applications 313 can run on each VM 305.

As illustrated in FIG. 3, a hypervisor runs on each physical hostcomputer 210. In one embodiment, the hypervisor is in the form of ahardware level or “bare metal” hypervisor 307, and acts as thefunctioning operating system for the VMs 305 and any additionalvirtualization environment software. As used herein, the term “hardwarelevel hypervisor” means a component that acts as an interface between atleast one VM 305 and the hardware of the host computer 210 (as opposedto an operating system running on the host) for at least some services.Note that services described herein as being performed by a hypervisor307 are, under certain virtualization scenarios, performed by acomponent with a different name, such as “supervisor virtual machine,”“virtual machine manager (VMM),” “service partition,” or “domain 0(dom0).”

In one embodiment, the virtualization environment 311 is in the form ofsoftware provided by VMware, Inc. In this case, the hypervisor 307 is inthe form of VMware's hardware level hypervisor VMware ESX 307. It is tobe understood that the name of VMware's hardware level hypervisor 307can change between product releases (for example, it used to be calledESX Server and in the future could be called something else). In aVMware based virtualization environment 311, the supporting softwaresuite can be VMware vSphere, which is a VMware cloud enabledvirtualization software package. VMware vSphere runs on top of ESX. Itis to be understood that the name of VMware's cloud enabledvirtualization software package can change between product releases. Itis to be further understood that although VMware virtualizationenvironments 311 are discussed herein, other embodiments can beimplemented in the context of other virtualization environments 311 thatprovide similar functionality and features. For example, in otherembodiments virtualization environments such as Microsoft's Hyper-V areused.

Note that although the shared storage 309 utilized by the cluster isillustrated and described in conjunction with FIG. 3 as if it were asingle storage device 160, in practice it is typically implemented withmultiple underlying physical storage devices 160, which are managed bythe clustering and storage system 301 so as to appear as a singlestorage device 160 to computer systems 210 accessing the shared storage309. An example of the architecture of one such scenario according toone embodiment is illustrated in FIG. 4, which is discussed in moredetail below. In some embodiments, the shared storage 309 is in the formof virtual storage provided by the virtualization environment 311, suchas a set of one or more VMware disk(s) in the virtual machine disk(VMDK) format. An embodiment of this type is described in more detailbelow. In some embodiments, the cluster 300 is implemented using VeritasStorage Foundation High Availability (SFHA) or Veritas StorageFoundation Cluster File System High Availability (SFCFSHA), althoughother embodiments can be implemented in the context of other clusteringand storage management environments, such as Microsoft Cluster Server.

As explained in greater detail below in conjunction with FIG. 5, thequality of service manager 101 optimizes the quality of service toprioritized VMs 305 and applications 313 by taking into account thevarying quality of shared computing resources 315, such as CPU, memory,I/O bandwidth, I/O channels, storage, performance boosting cache,replication links, etc. As the term is used herein, a “shared computingresource” is any physical or logical resource that is shared amongcomponents on the computer system 210 in order to facilitatevirtualization. The quality of service manager 101 evaluates the sharedresources 315, and leverages distinctions in quality between resources315 of the same type to provide quality of service to the VMs 305 basedon their priority 507. In other words, multiple VMs 305 (or theapplications 313 running thereon) running on a single host 210 andmaking requests for shared computing resources 315 to the hypervisor 307thereon can have different priorities 507, based on, for example, theservice level agreements entered into with different customers orinternal clients being guaranteed varying levels of availability.Conventional hypervisors 307 fail to recognize that not all sharedcomputing resources 315 of the same type behave in same way at allpoints in time. In fact, shared resources 315 of the same type (e.g.,multiple I/O channels) can greatly vary in quality (e.g., availability,bandwidth, reliability, speed). The quality of service manager 101 makesoptimum use of the shared resources 315 by providing the best availableresources 315 to the VMs 305 and/or applications 313 with the highestpriorities 507. In different embodiments, this functionality can be usedto identify distinctions in quality among any type of shared computingresource 315, and provide varying qualities of service to specific VMs305 (or applications 313) based on their priority 507.

FIG. 3 shows the quality of service manager 101 running at a hypervisor307 level and providing quality of service that takes into account thevarying quality of shared computing resources 315 assigned to VMs 305 bythe hypervisor 307. It is to be understood that in other embodiments,the quality of service manager 101 runs at the level of a componentother than a hypervisor 307 and assigns shared computing resources 315to components other than VMs 305 in order to facilitate other forms ofvirtualization. For example, as described in detail below in conjunctionwith FIG. 6, in one embodiment the quality of service manager 101 runsat a multipathing component 601/HBA 235 level, and assigns sharedresources 315 to applications 313 attempting to access a virtualmultipathed disk 603. For purposes of this description, it can behelpful to recall that a VM 305 is a specialized type of application313, and to think of VMs 305 this way.

FIG. 4 illustrates an example of a shared storage 309 architecture,according to some embodiments. Intelligent storage arrays 190 groupmultiple physical storage devices 160 into single logical units (LUNs)401. Each LUN 401 appears as a single storage device 160 to an accessinghost 210 or guest 305. The logical volume manager 403 of the clusteringand storage system 301 can in turn provide software level logical volumemanagement to create multiple logical volumes 405 on top of each LUN401. Although FIG. 4 illustrates two LUNs 401 each of which is logicallytreated as three volumes 405, it is to be understood that clusters 300can contain more or fewer LUNs 401, and that more or fewer logicalvolumes 405 can be built on top of a given LUN 401.

Turning to FIG. 5, the modules of the quality of service manager 101 areillustrated in more detail according to one embodiment. A quality ratingassigning module 501 of the quality of service manager 101 assignsquality ratings 503 to each shared resource 315 that can be utilized bya VM 305 on the host 210. The collection of such computing resources 315can be thought of as a shared computing resource pool 505. The factorsthat affect the quality of a computing resource 315 can be static (e.g.,the average access speed of a storage device or the maximum throughputof a network connection as determined by its underlying hardwareconfiguration) or dynamic (e.g., the current amount of available spaceon a storage device or the current load on a communication channel). Indifferent embodiments, the quality rating assigning module 501 canassign quality ratings 503 to computing resources 315 based on staticfactors, dynamic factors or a combination of the two. Because thequality of computing resources 315 can change dynamically, the qualityrating assigning module 501 can continuously or periodically evaluatethe resources 315 in the pool 505, and determine current quality ratings503 based on their current status (e.g., based on real time changes tomonitored capacities of the computing resources 315 that vary as afunction of, e.g., level or type of usage). New computing resources 315can be added to the pool 505, for example if a new network adapter isadded to the computing device 210. When this occurs, the quality ratingassigning module 501 detects the new computing resource 315 and assignsit an appropriate quality rating 503.

It is to be understood that the quality of shared computing resourcescan be a function of its programmatic configuration, instead of or inaddition to the characteristics of any underlying hardware. For example,in one embodiment described in greater detail below in conjunction withFIG. 6, multiple priority queues 607 are maintained for each host busadapter (HBA) 235. The I/Os in the higher priority queues 607 are sentto the HBA 235 at a higher frequency than those in lower priority queues607, and thus the quality rating assigning module 501 assigns higherquality ratings 503 to the higher priority queues 607. However, thefrequency with which each queue 607 is processed is a function of how itis managed programmatically, as opposed to any inherent characteristicsof the HBA 235 or other underlying components.

As the term is used herein, a quality rating 503 is a quantification ofthe quality of a shared computing resource 315. Different internalformats can be used to represent quality ratings 503 in differentembodiments (e.g., numbers on a scale, alphanumeric descriptors,percentages, etc.). In some embodiments, quality ratings 503 quantifydifferent qualitative factors for different types of resources 315(e.g., capacity for storage devices, bandwidth for I/O channels). Insome embodiments, multiple qualitative factors are quantified forindividual resources 315 by a single quality rating 503 (e.g., capacity,bandwidth and latency, represented by, for example, separate fields in aquality rating object). In other embodiments, a quality rating 503 is inthe form of a single quantification of a resource's overall quality.

The different VMs 305 have different assigned priorities 507, which canbe based on the corresponding service level agreements of theapplications 313 running thereon, or on other factors that determine theVM's priority 507 relative to that of the other VMs 305 running on thesame host 210. In one embodiment, the priorities 507 are assigned to theVMs 305, and applications 313 are run on VMs 305 with priorities 507corresponding to the level of service to be provided to the specificapplication 313. In another embodiment, the priorities 507 are assignedto the applications 313 themselves, and each VM 305 takes its priority507 from that of the application 313 that runs thereon. As described ingreater detail below in conjunction with FIG. 6, in other embodimentspriorities 507 can be assigned to each one of multiple applications 313that run on one or more physical computer(s) 210, without considerationof VMs 305. Priorities 507 can be assigned to VMs 305 and/orapplications 313 by administrators or other users (e.g., through a userinterface) based on quality of service guaranties or other factors. Forexample, business critical applications can be run on a particular VMwith a high priority 507. As the term is used herein, a priority 507 isan indicator of a quality of service to be provided to a VM 305 orapplication 313 relative to other VMs 305 or applications 313 on thesame host or otherwise similarly situated. Different internal formatscan be used to represent priorities 507 in different embodiments (e.g.,low-medium-high, numbers on a scale, etc.). The priorities 507 arestored (e.g., by the clustering and storage system 301) in an accessibleformat (e.g., a global data structure, a database, etc.) such that thequality of service manager 101 can access them.

A request receiving module 509 of the quality of service manager 101receives requests 511 made to the hypervisor 307 for computing resources315. Recall that the quality of service manager 101 runs at thehypervisor 307 level. Therefore, the request receiving module 509 canreceive the requests 511 of interest made to the hypervisor 307 byintercepting or otherwise filtering calls made to the hypervisor 307,and identifying those that request shared computing resources 315.

A priority identifying module 513 of the quality of service manager 101identifies the priority 507 of the VM 305 (or application 313) that madethe request 511 for the shared resource 315. In one embodiment, requests511 for shared resources are tagged with an identifier of the originator(e.g., the ID of the VM 305 that made the request 511). In this case,the priority identifying module 513 identifies the originator of therequest 511 from the tag, and retrieves the corresponding priority 507,e.g., from the global data structure. In other embodiments, the priorityidentifying module 513 identifies the originator of the request 511 (andhence is able to look up and retrieve its priority 507) in other ways.For example, where the request 511 is in the form of an attempt toaccess shared storage media 309, the priority identifying module 513 canidentify the originator of the request 511 by determining the LUN 401 onwhich the attempted I/O operation is occurring.

In response to requests 511 for shared computing resources 315, aresource assigning module 515 of the quality of service manager 101assigns specific shared resources 315 of the type requested from thepool 505, based on priority 507 of the requester (i.e., the VM 305 orapplication 313) and the quality rating 503 of the resource 315. In oneembodiment, this process can comprise assigning the resources 315 withhigher quality ratings 503 to service requests 511 made by componentswith higher priorities 507. In other embodiments, more specific levelsof granularity are used to make the assignments. For example, prioritiescan indicate specific factors of importance such as reliability, speed,bandwidth, etc., and shared resources 315 having varying quality ratings503 concerning these specific factors can be assigned to VMs 305 withcorresponding factor-specific priorities 507. The exact levels ofgranularity to use for both quality ratings 503 and priorities 507, bothgenerally and concerning specific factors, can vary between embodimentsas desired. Likewise, the exact logic to use to assign resources 315 torequesters based on the mapping between quality ratings 503 andpriorities 507 can vary between embodiments. By assigning sharedcomputing resources 315 to VMs 305 and applications 313 based on thecorrespondence between quality ratings 503 and priorities 507, thequality of service manager 101 can provide quality of service tospecific components in line with their associated specific priorities507, as determined by service level agreement or otherwise. This makesmore judicious use of the common pool 505 of shared resources 315.

In some embodiments, a logging module 517 of the quality of servicemanager 101 monitors the usage of shared computing resources 315 byspecific VMs 305 (or applications 313) over time, and writescorresponding information to a log 519 for future reference. The loggingmodule 517 can log information of varying levels of detail in differentembodiments as desired. For example, the log 519 can document basicaudit/statistical resource usage information, or the actual valuesutilized in accessing specific resources 315, such as the offsets of I/Osent to a given path, storage device, replication link etc., e.g., atthe level of a file change log. The logged information can be used, forexample, to compute incremental data updates (e.g., incremental datasent on a given pipe) and provide it to an off-host processing service(not illustrated), such as an incremental backup system or the like.

Drawing attention back to FIG. 3, a specific use case is now describedto help illustrate the operation of the quality of service manager 101according to one embodiment. As noted above, in one embodiment, thevirtualization environment 311 is in the form of the VMware vSphereplatform, and the hypervisor 307 is VMware's hardware level hypervisorESX. In this embodiment, a plurality of VMs 305 run on top an instanceof ESX 307 on a given host computer 210, and make requests 511 to ESX307 to read and write data to and from a VMDK 309. (A VMDK 309 is avirtual machine disk in a VMware specific format.) The resourceassigning module 515 assigns specific ones of these I/O operations toI/O queues 607 (which are a type of shared resource 315) of varyingpriority levels (i.e., varying quality ratings 503) based on thepriority 507 of the VM 305 that requested the I/O operation. Note thatI/O queues 607 are not specifically illustrated in FIG. 3, but FIG. 6illustrates shared computing resources 315 in the form of queues 607having different levels of priority. In the example of the specific usecase being described, each VM 305 running on the host 210 has one ofthree priorities: high, medium or low. The resource assigning module 515assigns one of the three I/O queues 607 to service I/O requests 511 fromthe VMs 305, by placing the I/O requests 511 into the assigned queue607. The three queues 607 have quality ratings 503 of high, medium andlow respectively. I/O requests 511 originating from VMs 305 with apriority 307 of high are placed in the high rated queue 607, thoseoriginating from medium priority VMs 305 in the medium rated queue 607and those from low rated VMs 305 in the low rated queue 607. The queues607 are serviced according to their rating, such that I/O requests 511from high priority VMs 305 are processed with the highest quality ofservice and so on. It is to be understood that in similar use cases,more than three levels of quality rating 503 and priority 507 can beutilized.

In the use case being described, the priority identifying module 513determines the VM 305 from which a given I/O request 511 originated (andhence its priority 507) by determining to or from which underlying LUN401 the request 511 is directed. Recall that in this use case the sharedstorage 309 is in the form of a VMDK virtual disk. Thus, to determinethe target LUN 401, the quality of service manager 101 creates a mapping317 of the VMDK disk blocks affected by the I/O operation and thecorresponding offsets in the set of LUNs 401 underlying the VMDK 309.This mapping 317 indicates the target LUN 401, and hence the originatingVM 305 and its priority 507.

Different methodologies can be used to determine the VMDK disk 309 toLUN 401 mapping 317 in different implementations of this use case. Forexample, in one implementation VMware web-services APIs are used todetermine the set of LUNs 401 which are part of a given VMwaredatastore. The storage mapping is determined using a given VMwarecommand with a specific command line option (currently “vmkfstools −t0”although the command and calling parameter(s) could change in futureversions of VMware products). This command outputs the mapping for VMDKblocks to offsets in a set of universally unique identifiers (UUIDs).Note that these UUIDs do not directly correlate to actual storage LUNs401, but are stored in individual storage LUNs 401 beginning at a fixedoffset (currently offset 00100080 in VMware Virtual Machine File System4.1). By reading data at this offset on devices which are part of thegiven VMware datastore (as determined via the web-service APIs asdescribed above), it is determined which LUNs 401 have which givenUUIDs. Because the quality of service manager 101 has have the mappingfor VMDK blocks to offsets in UUIDs as returned by the “vmkfstools −t0”command, and has determined which LUN 401 has which UUID, the quality ofservice manager 101 can now construct a mapping 317 of VMDK blocks toLUNs 401. Note that in other implementations, this mapping 317 isobtained other ways, for example by using certain VMware APIs whereavailable. In any case, the mapping 317 indicates which blocks of VMDKdata reside in which offset of a given LUN 401. Based on this mapping317, a multipathing component (e.g., VxDMP in a VMware ESX environment)can determine which VM 305 a given I/O packet is coming from or goingto, and hence which VM 305 made the request 511. Multipathing isdiscussed in more detail below in conjunction with FIG. 6. Once the VM305 that made the request 511 is identified, the priority identifyingmodule 513 identifies the priority 507 of that VM 305, and the resourceassigning module 515 assigns the specific I/O operation to an I/O queue607 with a matching quality rating 503.

Turning now to FIG. 6, another use case for the quality of servicemanager 101 is described. In the use case illustrated in FIG. 6, ratherthan run at a hypervisor 307 level, the quality of service manager 101runs at the level of a multipathing component 601 of the clustering andstorage system 301, with some modules running at HBA 235 level. Morespecifically, some storage management products such as Veritas VolumeManager's Dynamic Multipathing component support multipathed storagedevices 603. A multipathed storage device 603 is a virtual disk which ismade available to initiators of I/O, wherein multiple physical paths 605exist between the virtual multipathed disk 603 and the underlyingphysical storage 160. Although FIG. 6 illustrates two paths 605 tophysical storage 160, in practice more (or fewer) paths 605 can exist.Additionally, although FIG. 6 illustrates the physical storage 160 as asingle component, typically the physical storage 160 would beimplemented with multiple underlying physical storage devices 160, forexample as illustrated in FIG. 4.

As illustrated in FIG. 6, multiple applications 313 run in the systemmemory 217 of a computer system 210, and initiate I/O operationstargeting the virtual multipathed disk 603. In response, themultipathing component 601 determines which path 605 to use for givenI/O operations. Conventionally, the multipathing component 601 employs aconventional load balancing policy (e.g., round robin, minimumq, etc.)to decide down which path 605 to send each I/O, and dynamically switchesbetween paths 605 if a given path 605 fails. In order to send an I/Odown a given path 605 to the physical storage 160, the multipathingcomponent 601 sends the I/O to a specific host adapter bus (HBA) 235configured to connect to the physical storage 160 via the given path605.

In one use case illustrated in FIG. 6, the quality of service manager101 maintains multiple priority queues 607 for each HBA 235. Themultipathing component 601 receives I/Os targeting the virtualmultipathed disk 603, and uses its load balancing policy to assign I/Osto specific HBAs 235. At the HBA 235 level, the resource assigningmodule 515 then assigns given I/Os to specific priority queues 607 forthe specific HBA 235, based on the quality rating 503 of the queue 607and the priority 507 of the application 313 which initiated the I/Ooperation. More specifically, FIG. 6 illustrates three priority queues607 per HBA 235, one each for I/Os from high priority applications 313,medium priority applications 313 and low priority applications 313. Inother embodiments, more, fewer or differently rated queues 607 can bemaintained per HBA 235 as desired.

The priority identifying module 513 identifies the application 313 fromwhich the I/O originated (for example, from a tag in the I/O request 511or the LUN 401 on which the I/O operation is occurring), and henceidentifies the application's priority 507. The resource assigning module515 inserts the I/O in the appropriate priority queue 607, based on thepriority 507 of the application 313. For example, in the illustratedthree queue 607 embodiment, I/Os originating from applications 313 witha priority 507 of high are inserted in high priority queue 607, thosefrom medium priority applications 313 in the medium priority queue 607and those from the low priority applications 313 in the low priorityqueue 607.

The quality of service manager 101 sends I/Os down each HBA 235 from itspriority queues 607 based on their respective priorities. In otherwords, the quality of service manager 101 selects the most I/Os from thehigh priority queue 607, fewer I/Os from the middle priority queue 607and the least number of I/Os from low priority queue 607. The specificproportion of I/Os to select from each queue 607 can be determined basedon relative priority, or can be set by an administrator or other user.The exact proportion to use is a variable design parameter, anddifferent proportions can be used in different embodiments as desired.This servicing of I/O operations in proportion to the priority 507 oftheir originating applications 313 provides a higher level of service tohigher priority applications 313.

In another embodiment of the use case illustrated in FIG. 6, qualityratings 503 are assigned to the specific paths 605 themselves, asopposed or in addition to the queues 607 to the paths 605.Conventionally, users can assign different weights to different paths605 of a multipathed storage device 603. The weights are then used as ascheduling mechanism, such that each weight determines the proportion ofI/Os being sent to the corresponding path 605. Weights can also bedetermined on the fly, based on factors such as the bandwidth of thepath 605, the number of hops to the physical storage device 160, thestability of the path 605, etc. In this embodiment, the quality ratingassigning module 501 assigns quality ratings 503 to the paths 605 basedon their weights (e.g., the higher the weight the higher the assignedquality rating 503). In some embodiments, rather than basing the qualityratings 503 of paths 605 on their assigned weights, the quality ratingassigning module 501 actively determines a quality rating 503 for eachpath 605, by actively measuring factors such as its reliability, speed,availability, etc. To measure such factors, the quality rating assigningmodule 501 can statically or dynamically monitor factors such as numberof hops, stability of the path 605, failures encountered in the past,storage controller upgrades, etc. In either case, the priorityidentifying module 513 then identifies the priorities 507 ofapplications 313 that send I/Os to the multipathed storage device 603,and the resource assigning module 515 sends each I/O down a path 605with a corresponding quality rating 501. For example, I/Os originatingfrom high priority applications are sent down the path 605 with thehighest quality rating 503, etc. It is to be understood that in someembodiments, the assignment of quality ratings 503 to paths 605 and theassignment of quality ratings to queues 607 are utilized separately inorder to provide quality of service to applications 313 based on theirpriority 507. In other embodiments, these features are used incombination. In these embodiments, each I/O is assigned to a specificpath 605 with a specific quality rating 503 based on the priority 507 ofthe originating application 313, and then further assigned to specificpriority queue 607 for that path 605, again based on the originatingapplication's priority 507 (e.g., at a different level of granularity).

To apply the use case of FIG. 6 to a host running multiple VMs 305,priorities 507 can be assigned to VMs 305 as described above, andspecific applications 313 can be run on a VM 305 with an appropriatepriority 507.

As will be understood by those familiar with the art, the invention maybe embodied in other specific forms without departing from the spirit oressential characteristics thereof. Likewise, the particular naming anddivision of the portions, modules, agents, managers, components,functions, procedures, actions, layers, features, attributes,methodologies, data structures and other aspects are not mandatory orsignificant, and the mechanisms that implement the invention or itsfeatures may have different names, divisions and/or formats. Theforegoing description, for purpose of explanation, has been describedwith reference to specific embodiments. However, the illustrativediscussions above are not intended to be exhaustive or limiting to theprecise forms disclosed. Many modifications and variations are possiblein view of the above teachings. The embodiments were chosen anddescribed in order to best explain relevant principles and theirpractical applications, to thereby enable others skilled in the art tobest utilize various embodiments with or without various modificationsas may be suited to the particular use contemplated.

1. (canceled)
 2. A computer-implemented method comprising: assigning acurrent quality rating to a shared computing resource, wherein thecurrent quality rating of the shared computing resource quantifies aplurality of factors related to the shared computing resource, and theshared computing resource is accessible by a plurality of virtualmachines; receiving a request for the shared computing resource, whereinthe request is received from a requesting virtual machine (VM) fromamong the plurality of virtual machines; and assigning the sharedcomputing resource to the requesting VM, wherein the requesting virtualmachine is associated with a priority rating, and the assigning is basedon the priority rating of the requesting VM and the quality rating ofthe shared computing resource.
 3. The computer-implemented method ofclaim 2, wherein the plurality of shared computing resources comprisesat least one of memory, storage capacity, latency, and input/output(I/O) bandwidth.
 4. The computer-implemented method of claim 2, whereinthe priority rating of the requesting VM is based, at least in part, ona service level agreement (SLA), and the SLA specifies a minimum levelof service associated with the requesting VM.
 5. Thecomputer-implemented method of claim 2, wherein the plurality of factorsrelated to the shared computing resource comprises a static factor. 6.The computer-implemented method of claim 2, wherein the plurality offactors related to the shared computing resource comprises a dynamicfactor.
 7. The computer-implemented method of claim 6, furthercomprising: periodically evaluating the shared computing resource,wherein evaluating the shared computing resource comprises determiningwhether the dynamic factor has changed since a previous evaluation ofthe dynamic factor; in response to determining that the dynamic factorhas changed since the previous evaluation, updating the current qualityrating for the shared computing resource; and subsequent to updating thecurrent quality rating for the shared computing resource, assigning adifferent shared computing resource to the requesting VM, wherein thedifferent shared computing resource is assigned to the requesting VM inorder to maintain a minimum level of service associated with therequesting VM.
 8. The computer-implemented method of claim 7, whereinthe specific level of service associated with the requesting VM is basedon a service level agreement (SLA).
 9. The computer-implemented methodof claim 2, wherein the shared computing resource is one of a pluralityof shared computing resources that are accessible by the requesting VM.10. The computer-implemented method of claim 9, further comprising:adding a new computing resource to the plurality of shared computingresources; assigning a current quality rating to the new computingresource, wherein the current quality rating of the new computingresource quantifies a plurality of factors related to the new computingresource; determining whether the new computing resource could maintaina level of service associated with the requesting VM more accuratelythan the shared computing resource that was previously assigned to therequesting VM; and in response to determining that the new computingresource could more accurately maintain the level of service associatedwith the requesting VM, assigning the new computing resource to therequesting VM.
 11. A non-transitory computer-readable storage mediumcomprising program instructions executable to perform a methodcomprising: assigning a current quality rating to a shared computingresource, wherein the current quality rating of the shared computingresource quantifies a plurality of factors related to the sharedcomputing resource, and the shared computing resource is accessible by aplurality of virtual machines; receiving a request for the sharedcomputing resource, wherein the request is received from a requestingvirtual machine (VM) from among the plurality of virtual machines; andassigning the shared computing resource to the requesting VM, whereinthe requesting virtual machine is associated with a priority rating, andthe assigning is based on the priority rating of the requesting VM andthe quality rating of the shared computing resource.
 12. Thenon-transitory computer-readable storage medium of claim 11, wherein thepriority rating of the requesting VM is based, at least in part, on aservice level agreement (SLA), and the SLA specifies a minimum level ofservice associated with the requesting VM.
 13. The non-transitorycomputer-readable storage medium of claim 11, wherein the plurality offactors related to the shared computing resource comprises at least oneof: a static factor, and a dynamic factor.
 14. The non-transitorycomputer-readable storage medium of claim 13, wherein the method furthercomprises: periodically evaluating the shared computing resource,wherein evaluating the shared computing resource comprises determiningwhether the dynamic factor has changed since a previous evaluation ofthe dynamic factor; in response to determining that the dynamic factorhas changed since the previous evaluation, updating the current qualityrating for the shared computing resource; and subsequent to updating thecurrent quality rating for the shared computing resource, assigning adifferent shared computing resource to the requesting VM, wherein thedifferent shared computing resource is assigned to the requesting VM inorder to maintain a minimum level of service associated with therequesting VM.
 15. The non-transitory computer-readable storage mediumof claim 14, wherein the specific level of service associated with therequesting VM is based on a service level agreement (SLA).
 16. Thenon-transitory computer-readable storage medium of claim 11, wherein themethod further comprises: adding a new computing resource to a pluralityof shared computing resources, wherein the plurality of shared computingresources comprises the shared computing resource and the new computingresource; assigning a current quality rating to the new computingresource, wherein the current quality rating of the new computingresource quantifies a plurality of factors related to the new computingresource; determining whether the new computing resource could maintaina level of service associated with the requesting VM more accuratelythan the shared computing resource that was previously assigned to therequesting VM; and in response to determining that the new computingresource could more accurately maintain the level of service associatedwith the requesting VM, assigning the new computing resource to therequesting VM.
 17. A system comprising: one or more processors; and amemory coupled to the one or more processors, wherein the memory storesprogram instructions executable to perform a method comprising:assigning a current quality rating to a shared computing resource,wherein the current quality rating of the shared computing resourcequantifies a plurality of factors related to the shared computingresource, and the shared computing resource is accessible by a pluralityof virtual machines; receiving a request for the shared computingresource, wherein the request is received from a requesting virtualmachine (VM) from among the plurality of virtual machines; and assigningthe shared computing resource to the requesting VM, wherein therequesting virtual machine is associated with a priority rating, and theassigning is based on the priority rating of the requesting VM and thequality rating of the shared computing resource.
 18. The system of claim17, wherein the priority rating of the requesting VM is based, at leastin part, on a service level agreement (SLA), and the SLA specifies aminimum level of service associated with the requesting VM.
 19. Thesystem of claim 17, wherein the plurality of factors related to theshared computing resource comprises at least one of: a static factor,and a dynamic factor.
 20. The system of claim 19, wherein the methodfurther comprises: periodically evaluating the shared computingresource, wherein evaluating the shared computing resource comprisesdetermining whether the dynamic factor has changed since a previousevaluation of the dynamic factor; in response to determining that thedynamic factor has changed since the previous evaluation, updating thecurrent quality rating for the shared computing resource; and subsequentto updating the current quality rating for the shared computingresource, assigning a different shared computing resource to therequesting VM, wherein the different shared computing resource isassigned to the requesting VM in order to maintain a minimum level ofservice associated with the requesting VM, and the specific level ofservice associated with the requesting VM is based on a service levelagreement (SLA).
 21. The system of claim 20, wherein the method furthercomprises: adding a new computing resource to the plurality of sharedcomputing resources; assigning a current quality rating to the newcomputing resource, wherein the current quality rating of the newcomputing resource quantifies a plurality of factors related to the newcomputing resource; determining whether the new computing resource couldmaintain a level of service associated with the requesting VM moreaccurately than the shared computing resource that was previouslyassigned to the requesting VM; and in response to determining that thenew computing resource could more accurately maintain the level ofservice associated with the requesting VM, assigning the new computingresource to the requesting VM.